

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>分级缓存 &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/jquery.js"></script>
        <script src="../../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../../" id="documentation_options" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/doctools.js"></script>
        <script src="../../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../genindex/" />
    <link rel="search" title="Search" href="../../../search/" />
    <link rel="next" title="归置组" href="../placement-groups/" />
    <link rel="prev" title="CLAY 码插件" href="../erasure-code-clay/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../../">Ceph 存储集群</a></li>
          <li class="breadcrumb-item"><a href="../">集群运维</a></li>
      <li class="breadcrumb-item active">分级缓存</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../../_sources/rados/operations/cache-tiering.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephadm/">Cephadm</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../">Ceph 存储集群</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../configuration/">配置</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">运维</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../operating/">操纵集群</a></li>
<li class="toctree-l3"><a class="reference internal" href="../health-checks/">健康检查</a></li>
<li class="toctree-l3"><a class="reference internal" href="../monitoring/">监控集群</a></li>
<li class="toctree-l3"><a class="reference internal" href="../monitoring-osd-pg/">监控 OSD 和归置组</a></li>
<li class="toctree-l3"><a class="reference internal" href="../user-management/">用户管理</a></li>
<li class="toctree-l3"><a class="reference internal" href="../pgcalc/">PG Calc</a></li>
<li class="toctree-l3"><a class="reference internal" href="../data-placement/">数据归置概览</a></li>
<li class="toctree-l3"><a class="reference internal" href="../pools/">存储池</a></li>
<li class="toctree-l3"><a class="reference internal" href="../erasure-code/">纠删码</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">分级缓存</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#id2">一些忠告</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id5">配置存储池</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id8">创建缓存层</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id9">配置缓存层</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id15">拆除缓存层</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id18">排查找不到的对象</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../placement-groups/">归置组</a></li>
<li class="toctree-l3"><a class="reference internal" href="../upmap/">使用 pg-upmap</a></li>
<li class="toctree-l3"><a class="reference internal" href="../read-balancer/">Operating the Read (Primary) Balancer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../balancer/">均衡器模块</a></li>
<li class="toctree-l3"><a class="reference internal" href="../crush-map/">CRUSH 图</a></li>
<li class="toctree-l3"><a class="reference internal" href="../crush-map-edits/">手动编辑一个 CRUSH 图</a></li>
<li class="toctree-l3"><a class="reference internal" href="../stretch-mode/">Stretch Clusters</a></li>
<li class="toctree-l3"><a class="reference internal" href="../change-mon-elections/">Configuring Monitor Election Strategies</a></li>
<li class="toctree-l3"><a class="reference internal" href="../add-or-rm-osds/">增加/删除 OSD</a></li>
<li class="toctree-l3"><a class="reference internal" href="../add-or-rm-mons/">增加/删除监视器</a></li>
<li class="toctree-l3"><a class="reference internal" href="../devices/">设备管理</a></li>
<li class="toctree-l3"><a class="reference internal" href="../bluestore-migration/">迁移到 BlueStore</a></li>
<li class="toctree-l3"><a class="reference internal" href="../control/">命令参考</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/community/">Ceph 社区</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/troubleshooting-mon/">监视器故障排除</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/troubleshooting-osd/">OSD 故障排除</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/troubleshooting-pg/">归置组排障</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/log-and-debug/">日志记录和调试</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/cpu-profiling/">CPU 剖析</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/memory-profiling/">内存剖析</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../man/">    手册页</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../troubleshooting/">故障排除</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../api/">APIs</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="id1">
<h1>分级缓存<a class="headerlink" href="#id1" title="Permalink to this heading"></a></h1>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Cache tiering has been deprecated in the Reef release as it
has lacked a maintainer for a very long time. This does not mean
it will be certainly removed, but we may choose to remove it
without much further notice.</p>
</div>
<p>分级缓存可提升后端存储内某些（热点）数据的 I/O 性能。
分级缓存需创建一个由高速而昂贵存储设备（如 SSD ）组成的存储池、
作为缓存层，以及一个相对低速/廉价设备组成的后端存储池
（或纠删码编码的）、作为经济存储层。
Ceph 的对象处理器决定往哪里存储对象，
分级代理决定何时把缓存内的对象刷回后端存储层；
所以缓存层和后端存储层对 Ceph 客户端来说是完全透明的。</p>
<p class="ditaa">
<img src="../../../_images/ditaa-948504c317d1f3a8c5a864e90d6209ee23899217.png"/>
</p>
<p>缓存层代理负责自动处理缓存层和后端存储之间的数据迁移。
然而，管理员仍可干预此迁移规则，
主要有两种场景：</p>
<ul>
<li><p><strong>writeback</strong> （回写）模式：管理员把缓存层配置为 <code class="docutils literal notranslate"><span class="pre">writeback</span></code> 模式时，
Ceph 客户端们会把数据写入缓存层、
并收到缓存层发来的 ACK ；随后，
写入缓存层的数据会被迁移到存储层、然后从缓存层刷掉。
直观地看，缓存层位于后端存储层的“前面”，</p>
<p>当 Ceph 客户端要读取的数据位于存储层时，
缓存层代理会把这些数据迁移到缓存层，
然后再发往 Ceph 客户端。从此， Ceph 客户端将与缓存层进行 I/O 操作，
直到数据不再被读写。
此模式对于易变数据来说较理想（如照片/视频编辑、事务数据等）。</p>
</li>
<li><p><strong>readproxy</strong> （读代理）模式：在这个模式下，
会直接用缓存层已有的对象，
可某一对象没在缓存中时，这个请求会被代理到底下一层。
这对于 <code class="docutils literal notranslate"><span class="pre">writeback</span></code> 模式的关闭很有用，
因为它能保证在抽掉缓存层的同时还确保业务可以正常运转，
而且不会新增对象到缓存层。</p></li>
</ul>
<p>其他缓存模式还有：</p>
<ul class="simple">
<li><p><strong>readonly</strong> 遇到读操作时把对象提升到缓存里；
写操作会被转发到基础层。
这个模式适合需要只读业务、而不要求系统保证强一致性的人。
（ <strong>警告</strong>：对象在基础层更新后，
Ceph <strong>不会</strong> 尝试把相应对象的更新给同步到缓存里。
这个模式还是实验性的，
所以启用它时必须加 <code class="docutils literal notranslate"><span class="pre">--yes-i-really-mean-it</span></code> 选项。）</p></li>
<li><p><strong>none</strong> 用于完全禁用缓存。</p></li>
</ul>
<section id="id2">
<h2>一些忠告<a class="headerlink" href="#id2" title="Permalink to this heading"></a></h2>
<p>分级缓存会使大多数工作载荷的<em>性能下降</em>。
用户使用此功能时应该格外小心。</p>
<ul class="simple">
<li><p><em>与业务载荷相关</em>：缓存是否能提升性能与业务载荷息息相关。
因为把对象迁入或迁出缓存是有代价的，所以，
当数据集的访问模式有<em>明显倾斜</em>时缓存才能高效地运行，
比如大多数请求只接触少数对象时。
缓存存储池应该足够大，
可以容纳业务载荷的常用部分，以避免抖动。</p></li>
<li><p><em>难以评估</em>：用户做过的大多数性能评估都表明，
使用分级缓存时性能非常差，
部分原因是很少有用户会让请求偏向于一小部分对象，
所以缓存需要很长时间来“预热”，
而预热的代价又很高。</p></li>
<li><p><em>通常更慢</em>：对于不适合用分级缓存的业务载荷，
启用分级缓存后性能反而比没缓存的常规 RADOS 存储池更慢。</p></li>
<li><p><em>librados 对象枚举</em>： librados 级的对象枚举 API
不是为这种场景设计的。
如果你的应用程序直接使用 librados 、并需要枚举对象，
分级缓存可能不会按预期方式运行。
（ RGW 、 RBD 、 CephFS 没有这样的问题。）</p></li>
<li><p><em>复杂性</em>：启用分级缓存意味着
RADOS 集群要引入很多额外机制和复杂性。
这样就增加了遇到系统缺陷的可能性，
而其他用户未必会遇到这个缺陷，
如此一来你的系统风险更高。</p></li>
</ul>
<section id="id3">
<h3>已知适用的场景<a class="headerlink" href="#id3" title="Permalink to this heading"></a></h3>
<ul class="simple">
<li><p><em>RGW 时域分布（ time-skewed ）</em>：
如果 RGW 业务载荷契合这样的场景就能良好地运行：
所有读操作都指向近期写入的对象们，
配置的分级缓存按配置的时间间隔、定期地把近期写入的对象从缓存迁入底层。</p></li>
</ul>
</section>
<section id="id4">
<h3>已知不适用的场景<a class="headerlink" href="#id4" title="Permalink to this heading"></a></h3>
<p>下面是<em>已知的与分级缓存不和睦</em>的配置。</p>
<ul class="simple">
<li><p><em>底层为纠删码、缓存为多副本存储池的 RBD 业务</em>：
这是常见需求，但是一般不能很好地运行。
即使是分布良好的业务载荷，
仍然会向冷对象发送很小的写请求，
然而纠删码存储池还不支持小的写入，
就算是一个很小的写入（通常是 4 KB ）也得把整个对象（通常为 4 MB 大）迁入缓存。
只有少数用户成功实施了这样的配置，也只有他们能用，
因为他们的数据极其冷门（备份）、而且不可能对性能敏感。</p></li>
<li><p><em>底层和缓存均为多副本存储池的 RBD 业务</em>：
RBD 的底层为多副本存储池时比纠删码好一些，
但是仍然高度依赖于业务载荷中的数量分布，
且难于验证。用户需要精通自己的业务载荷，
还得谨慎地调整分级缓存配置参数。</p></li>
</ul>
</section>
</section>
<section id="id5">
<h2>配置存储池<a class="headerlink" href="#id5" title="Permalink to this heading"></a></h2>
<p>要设置缓存层，你必须有两个存储池。一个作为后端存储、
另一个作为缓存。</p>
<section id="id6">
<h3>配置后端存储池<a class="headerlink" href="#id6" title="Permalink to this heading"></a></h3>
<p>设置后端存储池通常会遇到两种场景：</p>
<ul class="simple">
<li><p><strong>标准存储：</strong> 此时，Ceph存储集群内的存储池保存了一对象的多个副本；</p></li>
<li><p><strong>纠删存储池：</strong> 此时，存储池用纠删码高效地存储数据，
性能稍有损失。</p></li>
</ul>
<p>在标准存储场景中，你可以用 CRUSH 规则来标识失败域
（如 osd 、主机、机箱、机架、排等）。
当规则所涉及的所有驱动器规格、速度（转速和吞吐量）和类型相同时，
OSD 守护进程运行得最优。
创建规则集的详情见 <a class="reference external" href="../crush-map">CRUSH 图</a>。
创建好规则后，再创建后端存储池。</p>
<p>在纠删码编码情景中，创建存储池时指定好参数就会自动生成合适的规则集，
详情见<a class="reference external" href="../pools#create-a-pool">创建存储池</a>。</p>
<p>在后续例子中，我们把 <code class="docutils literal notranslate"><span class="pre">cold-storage</span></code> 当作后端存储池。</p>
</section>
<section id="id7">
<h3>配置缓存池<a class="headerlink" href="#id7" title="Permalink to this heading"></a></h3>
<p>缓存存储池的设置步骤大致与标准存储情景相同，但仍有不同：
缓存层所用的驱动器通常都是高性能的、且安装在专用服务器上、
有自己的 CRUSH 规则。
制定这样的规则时，要考虑到装有高性能驱动器的主机、
并忽略没有的主机。
详情见 <a class="reference internal" href="../crush-map-edits/#crush-map-device-class"><span class="std std-ref">CRUSH 设备类</span></a> 。</p>
<p>在后续例子中， <code class="docutils literal notranslate"><span class="pre">hot-storage</span></code> 作为缓存存储池、
<code class="docutils literal notranslate"><span class="pre">cold-storage</span></code> 作为后端存储池。</p>
<p>关于缓存层的配置及其默认值的详细解释请参考<a class="reference external" href="../pools#set-pool-values">存储池——调整存储池</a>。</p>
</section>
</section>
<section id="id8">
<h2>创建缓存层<a class="headerlink" href="#id8" title="Permalink to this heading"></a></h2>
<p>设置一缓存层需把缓存存储池挂接到后端存储池上：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt1:before {
  content: "$ ";
}
</style><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>add<span class="w"> </span><span class="o">{</span>storagepool<span class="o">}</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span></span>
</pre></div></div><p>例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>add<span class="w"> </span>cold-storage<span class="w"> </span>hot-storage</span>
</pre></div></div><p>用下列命令设置缓存模式：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>cache-mode<span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span><span class="o">{</span>cache-mode<span class="o">}</span></span>
</pre></div></div><p>例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>cache-mode<span class="w"> </span>hot-storage<span class="w"> </span>writeback</span>
</pre></div></div><p>缓存层盖在后端存储层之上，所以要多一步：
必须把所有客户端流量从存储池迁移到缓存存储池。
用此命令把客户端流量指向缓存存储池：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>set-overlay<span class="w"> </span><span class="o">{</span>storagepool<span class="o">}</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span></span>
</pre></div></div><p>例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>set-overlay<span class="w"> </span>cold-storage<span class="w"> </span>hot-storage</span>
</pre></div></div></section>
<section id="id9">
<h2>配置缓存层<a class="headerlink" href="#id9" title="Permalink to this heading"></a></h2>
<p>缓存层支持几个配置选项，
可按下列语法配置：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span><span class="o">{</span>key<span class="o">}</span><span class="w"> </span><span class="o">{</span>value<span class="o">}</span></span>
</pre></div></div><p>详情见<a class="reference external" href="../pools#set-pool-values">存储池——调整存储池</a>。</p>
<section id="id10">
<h3>目标尺寸和类型<a class="headerlink" href="#id10" title="Permalink to this heading"></a></h3>
<p>生产环境下，缓存层的 <code class="docutils literal notranslate"><span class="pre">hit_set_type</span></code> 用 <a class="reference external" href="https://en.wikipedia.org/wiki/Bloom_filter">Bloom 过滤器</a>：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>hit_set_type<span class="w"> </span>bloom</span>
</pre></div></div><p>例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span>hot-storage<span class="w"> </span>hit_set_type<span class="w"> </span>bloom</span>
</pre></div></div><p><code class="docutils literal notranslate"><span class="pre">hit_set_count</span></code> 和 <code class="docutils literal notranslate"><span class="pre">hit_set_period</span></code> 选项定义了要保留多少个这样的 HitSet ，
以及各 HitSet 覆盖的时间区间。</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>hit_set_count<span class="w"> </span><span class="m">12</span></span>
<span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>hit_set_period<span class="w"> </span><span class="m">14400</span></span>
<span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>target_max_bytes<span class="w"> </span><span class="m">1000000000000</span></span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p><code class="docutils literal notranslate"><span class="pre">hit_set_count</span></code> 设置得越大，
<code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code> 消耗的 RAM 越多。</p>
</div>
<p>保留一段时间以来的访问记录，
这样 Ceph 就能判断一客户端在一段时间内访问了某对象一次、
还是多次（存活期与热度）。</p>
<p><code class="docutils literal notranslate"><span class="pre">min_read_recency_for_promote</span></code> 定义了在处理一个对象的读操作时检查多少个 HitSet ，
检查结果将用于决定是否异步地提升对象。
它的取值应该在 0 和 <code class="docutils literal notranslate"><span class="pre">hit_set_count</span></code> 之间，如果设置为 0 ，
对象会一直被提升；如果设置为 1 ，就只检查当前 HitSet ，
如果此对象在当前 HitSet 里就提升它，否则就不提升；设置为其它值时，
就要挨个检查此数量的历史 HitSet ，如果此对象出现在
<code class="docutils literal notranslate"><span class="pre">min_read_recency_for_promote</span></code> 个 HitSet 里的任意一个，那就提升它。</p>
<p>还有一个相似的参数用于配置写操作，
它是 <code class="docutils literal notranslate"><span class="pre">min_write_recency_for_promote</span></code> 。</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>min_read_recency_for_promote<span class="w"> </span><span class="m">2</span></span>
<span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>min_write_recency_for_promote<span class="w"> </span><span class="m">2</span></span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>统计时间越长，
<code class="docutils literal notranslate"><span class="pre">min_read_recency_for_promote</span></code> 和 <code class="docutils literal notranslate"><span class="pre">min_write_recency_for_promote</span></code> 的值越高，
<code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code> 进程消耗的内存就越多，
特别是代理正忙着刷回或赶出对象时，
此时所有 <code class="docutils literal notranslate"><span class="pre">hit_set_count</span></code> 个 HitSet 都载入了内存。</p>
</div>
</section>
<section id="id11">
<h3>缓存空间消长<a class="headerlink" href="#id11" title="Permalink to this heading"></a></h3>
<p>缓存分层代理有两个主要功能：</p>
<ul class="simple">
<li><p><strong>刷回：</strong> 代理找出修改过（或脏）的对象、
并把它们转发给存储池做长期存储。</p></li>
<li><p><strong>赶出：</strong> 代理找出未修改（或干净）的对象、
并把最近未用过的赶出缓存。</p></li>
</ul>
<section id="id12">
<h4>绝对空间消长<a class="headerlink" href="#id12" title="Permalink to this heading"></a></h4>
<p>缓存分层代理可根据总字节数或对象数量来刷回或赶出对象，
用下列命令可指定最大字节数：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>target_max_bytes<span class="w"> </span><span class="o">{</span><span class="c1">#bytes}</span></span>
</pre></div></div><p>例如，用下列命令配置在达到 1TB 时刷回或赶出：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span>hot-storage<span class="w"> </span>target_max_bytes<span class="w"> </span><span class="m">1099511627776</span></span>
</pre></div></div><p>用下列命令指定缓存对象的最大数量：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>target_max_objects<span class="w"> </span><span class="o">{</span><span class="c1">#objects}</span></span>
</pre></div></div><p>例如，用下列命令配置对象数量达到 1M 时开始刷回或赶出：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span>hot-storage<span class="w"> </span>target_max_objects<span class="w"> </span><span class="m">1000000</span></span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>Ceph 不能自动确定缓存池的大小，所以这里必须配置绝对尺寸，
否则刷回、或赶出就不会生效。如果两个都配置了，
缓存分层代理会按先达到的阀值执行刷回或赶出。</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>只有在达到 <code class="docutils literal notranslate"><span class="pre">target_max_bytes</span></code> 或 <code class="docutils literal notranslate"><span class="pre">target_max_objects</span></code> 阀值时，
所有客户端的请求才会被阻塞。</p>
</div>
</section>
<section id="id13">
<h4>相对空间消长<a class="headerlink" href="#id13" title="Permalink to this heading"></a></h4>
<p>缓存分层代理可根据缓存存储池相对大小（<a class="reference external" href="#absolute-sizing">绝对空间消长</a>里由
<code class="docutils literal notranslate"><span class="pre">target_max_bytes</span></code> 和 <code class="docutils literal notranslate"><span class="pre">target_max_objects</span></code> 确定的）刷回或赶出对象。
当缓存池包含的已修改（或脏）对象达到一定比例时，
缓存分层代理就把它们刷回到存储池。用下列命令设置 <code class="docutils literal notranslate"><span class="pre">cache_target_dirty_ratio</span></code> ：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>cache_target_dirty_ratio<span class="w"> </span><span class="o">{</span><span class="m">0</span>.0..1.0<span class="o">}</span></span>
</pre></div></div><p>例如，设置为 <code class="docutils literal notranslate"><span class="pre">0.4</span></code> 时，脏对象达到缓存池容量的 40% 就开始刷回：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span>hot-storage<span class="w"> </span>cache_target_dirty_ratio<span class="w"> </span><span class="m">0</span>.4</span>
</pre></div></div><p>当脏对象达到其容量的一定比例时，要更快地刷回脏对象。用下列命令设置 <code class="docutils literal notranslate"><span class="pre">cache_target_dirty_high_rati</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>cache_target_dirty_high_ratio<span class="w"> </span><span class="o">{</span><span class="m">0</span>.0..1.0<span class="o">}</span></span>
</pre></div></div><p>例如，设置为 <code class="docutils literal notranslate"><span class="pre">0.6</span></code> 表示：脏对象达到缓存存储池容量的 60% 时，
将开始更激进地刷回脏对象。显然，
其值最好在 dirty_ratio 和 full_ratio 之间：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span>hot-storage<span class="w"> </span>cache_target_dirty_high_ratio<span class="w"> </span><span class="m">0</span>.6</span>
</pre></div></div><p>当缓存池利用率达到总容量的一定比例时，
缓存分层代理会赶出部分对象以维持空闲空间。
执行此命令设置 <code class="docutils literal notranslate"><span class="pre">cache_target_full_ratio</span></code> ：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>cache_target_full_ratio<span class="w"> </span><span class="o">{</span><span class="m">0</span>.0..1.0<span class="o">}</span></span>
</pre></div></div><p>例如，设置为 <code class="docutils literal notranslate"><span class="pre">0.8</span></code> 时，
干净对象占到总容量的 80% 就开始赶出缓存池：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span>hot-storage<span class="w"> </span>cache_target_full_ratio<span class="w"> </span><span class="m">0</span>.8</span>
</pre></div></div></section>
</section>
<section id="id14">
<h3>缓存时长<a class="headerlink" href="#id14" title="Permalink to this heading"></a></h3>
<p>你可以规定缓存层代理必须延迟多久才能把某个已修改（脏）对象刷回后端存储池：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>cache_min_flush_age<span class="w"> </span><span class="o">{</span><span class="c1">#seconds}</span></span>
</pre></div></div><p>例如，让已修改（或脏）对象需至少延迟 10 分钟才能刷回，
执行此命令：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span>hot-storage<span class="w"> </span>cache_min_flush_age<span class="w"> </span><span class="m">600</span></span>
</pre></div></div><p>你可以指定某对象在缓存层至少放置多长时间才能被赶出：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="o">{</span>cache-tier<span class="o">}</span><span class="w"> </span>cache_min_evict_age<span class="w"> </span><span class="o">{</span><span class="c1">#seconds}</span></span>
</pre></div></div><p>例如，要规定 30 分钟后才赶出对象，执行此命令：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span><span class="nb">set</span><span class="w"> </span>hot-storage<span class="w"> </span>cache_min_evict_age<span class="w"> </span><span class="m">1800</span></span>
</pre></div></div></section>
</section>
<section id="id15">
<h2>拆除缓存层<a class="headerlink" href="#id15" title="Permalink to this heading"></a></h2>
<p>回写缓存和只读缓存的去除过程不太一样。</p>
<section id="id16">
<h3>拆除只读缓存<a class="headerlink" href="#id16" title="Permalink to this heading"></a></h3>
<p>只读缓存不含变更数据，所以禁用它不会导致任何近期更改的数据丢失。</p>
<ol class="arabic">
<li><p>把缓存模式改为 <code class="docutils literal notranslate"><span class="pre">none</span></code> 即可禁用。</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>cache-mode<span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>none</span>
</pre></div></div><p>例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>cache-mode<span class="w"> </span>hot-storage<span class="w"> </span>none</span>
</pre></div></div></li>
<li><p>去除后端存储池的缓存池。</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>remove<span class="w"> </span><span class="o">{</span>storagepool<span class="o">}</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span></span>
</pre></div></div><p>例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>remove<span class="w"> </span>cold-storage<span class="w"> </span>hot-storage</span>
</pre></div></div></li>
</ol>
</section>
<section id="id17">
<h3>拆除回写缓存<a class="headerlink" href="#id17" title="Permalink to this heading"></a></h3>
<p>回写缓存可能含有更改过的数据，
所以在禁用并去除前，
必须采取些手段以免丢失缓存内近期更改的对象。</p>
<ol class="arabic">
<li><p>把缓存模式改为 <code class="docutils literal notranslate"><span class="pre">proxy</span></code> ，这样新的和更改过的对象将直接刷回到后端存储池。</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>cache-mode<span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>proxy</span>
</pre></div></div><p>例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>cache-mode<span class="w"> </span>hot-storage<span class="w"> </span>proxy</span>
</pre></div></div></li>
<li><p>确保缓存池已刷回，可能要等数分钟：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">rados<span class="w"> </span>-p<span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>ls</span>
</pre></div></div><p>如果缓存池还有对象，你可以手动刷回，例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">rados<span class="w"> </span>-p<span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span><span class="w"> </span>cache-flush-evict-all</span>
</pre></div></div></li>
<li><p>去除此盖子，这样客户端就不会被指到缓存了。</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>remove-overlay<span class="w"> </span><span class="o">{</span>storagetier<span class="o">}</span></span>
</pre></div></div><p>例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>remove-overlay<span class="w"> </span>cold-storage</span>
</pre></div></div></li>
<li><p>最后，从后端存储池剥离缓存层存储池。</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>remove<span class="w"> </span><span class="o">{</span>storagepool<span class="o">}</span><span class="w"> </span><span class="o">{</span>cachepool<span class="o">}</span></span>
</pre></div></div><p>例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>tier<span class="w"> </span>remove<span class="w"> </span>cold-storage<span class="w"> </span>hot-storage</span>
</pre></div></div></li>
</ol>
</section>
</section>
<section id="id18">
<h2>排查找不到的对象<a class="headerlink" href="#id18" title="Permalink to this heading"></a></h2>
<p>Under certain circumstances, restarting OSDs may result in unfound objects.</p>
<p>Here is an example of unfound objects appearing during an upgrade from Ceph
14.2.6 to Ceph 14.2.7:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mi">2</span><span class="o">/</span><span class="mi">543658058</span> <span class="n">objects</span> <span class="n">unfound</span> <span class="p">(</span><span class="mf">0.000</span><span class="o">%</span><span class="p">)</span>
<span class="n">pg</span> <span class="mf">19.12</span> <span class="n">has</span> <span class="mi">1</span> <span class="n">unfound</span> <span class="n">objects</span>
<span class="n">pg</span> <span class="mf">19.2</span><span class="n">d</span> <span class="n">has</span> <span class="mi">1</span> <span class="n">unfound</span> <span class="n">objects</span>

<span class="n">Possible</span> <span class="n">data</span> <span class="n">damage</span><span class="p">:</span> <span class="mi">2</span> <span class="n">pgs</span> <span class="n">recovery_unfound</span>
<span class="n">pg</span> <span class="mf">19.12</span> <span class="ow">is</span> <span class="n">active</span><span class="o">+</span><span class="n">recovery_unfound</span><span class="o">+</span><span class="n">undersized</span><span class="o">+</span><span class="n">degraded</span><span class="o">+</span><span class="n">remapped</span><span class="p">,</span> <span class="n">acting</span> <span class="p">[</span><span class="mi">299</span><span class="p">,</span><span class="mi">310</span><span class="p">],</span> <span class="mi">1</span> <span class="n">unfound</span>
<span class="n">pg</span> <span class="mf">19.2</span><span class="n">d</span> <span class="ow">is</span> <span class="n">active</span><span class="o">+</span><span class="n">recovery_unfound</span><span class="o">+</span><span class="n">undersized</span><span class="o">+</span><span class="n">degraded</span><span class="o">+</span><span class="n">remapped</span><span class="p">,</span> <span class="n">acting</span> <span class="p">[</span><span class="mi">290</span><span class="p">,</span><span class="mi">309</span><span class="p">],</span> <span class="mi">1</span> <span class="n">unfound</span>

<span class="c1"># ceph pg 19.12 list_unfound</span>
<span class="p">{</span>
    <span class="s2">&quot;num_missing&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
    <span class="s2">&quot;num_unfound&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
    <span class="s2">&quot;objects&quot;</span><span class="p">:</span> <span class="p">[</span>
        <span class="p">{</span>
            <span class="s2">&quot;oid&quot;</span><span class="p">:</span> <span class="p">{</span>
                <span class="s2">&quot;oid&quot;</span><span class="p">:</span> <span class="s2">&quot;hit_set_19.12_archive_2020-02-25 13:43:50.256316Z_2020-02-25 13:43:50.325825Z&quot;</span><span class="p">,</span>
                <span class="s2">&quot;key&quot;</span><span class="p">:</span> <span class="s2">&quot;&quot;</span><span class="p">,</span>
                <span class="s2">&quot;snapid&quot;</span><span class="p">:</span> <span class="o">-</span><span class="mi">2</span><span class="p">,</span>
                <span class="s2">&quot;hash&quot;</span><span class="p">:</span> <span class="mi">18</span><span class="p">,</span>
                <span class="s2">&quot;max&quot;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
                <span class="s2">&quot;pool&quot;</span><span class="p">:</span> <span class="mi">19</span><span class="p">,</span>
                <span class="s2">&quot;namespace&quot;</span><span class="p">:</span> <span class="s2">&quot;.ceph-internal&quot;</span>
            <span class="p">},</span>
            <span class="s2">&quot;need&quot;</span><span class="p">:</span> <span class="s2">&quot;3312398&#39;55868341&quot;</span><span class="p">,</span>
            <span class="s2">&quot;have&quot;</span><span class="p">:</span> <span class="s2">&quot;0&#39;0&quot;</span><span class="p">,</span>
            <span class="s2">&quot;flags&quot;</span><span class="p">:</span> <span class="s2">&quot;none&quot;</span><span class="p">,</span>
            <span class="s2">&quot;locations&quot;</span><span class="p">:</span> <span class="p">[]</span>
        <span class="p">}</span>
    <span class="p">],</span>
    <span class="s2">&quot;more&quot;</span><span class="p">:</span> <span class="n">false</span>
</pre></div>
</div>
<p>Some tests in the field indicate that the unfound objects can be deleted with
no adverse effects (see <a class="reference external" href="https://tracker.ceph.com/issues/44286#note-3">Tracker Issue #44286, Note 3</a>). Pawel Stefanski suggests
that deleting missing or unfound objects is safe as long as the objects are a
part of <code class="docutils literal notranslate"><span class="pre">.ceph-internal::hit_set_PGID_archive</span></code>.</p>
<p>Various members of the upstream Ceph community have reported in <a class="reference external" href="https://tracker.ceph.com/issues/44286">Tracker Issue
#44286</a> that the following versions of
Ceph have been affected by this issue:</p>
<ul class="simple">
<li><p>14.2.8</p></li>
<li><p>14.2.16</p></li>
<li><p>15.2.15</p></li>
<li><p>16.2.5</p></li>
<li><p>17.2.7</p></li>
</ul>
<p>See <a class="reference external" href="https://tracker.ceph.com/issues/44286">Tracker Issue #44286</a> for the
history of this issue.</p>
</section>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../erasure-code-clay/" class="btn btn-neutral float-left" title="CLAY 码插件" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../placement-groups/" class="btn btn-neutral float-right" title="归置组" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>